Field
The described embodiments relate to techniques for ingesting data from a variety of sources. More specifically, the described embodiments relate to techniques for ingesting data from a variety of sources using a dynamic data-ingestion pipeline.
Related Art
Big data poses a variety of challenges associated with the high volume of data being handled, including the need for significant data-processing computational capabilities and the need for new processing techniques. In addition, the coexistence of heterogeneous data sources can pose problems in computer systems that handle big data. For example, the heterogeneous data sources can make these computer systems more complicated, with separate data processing for the data from each data source. Furthermore, ingesting a large amount of data from a given data source, and then integrating this data at high velocity and with high quality is often difficult and expensive.
In the absence of solutions for these challenges, computer systems that handle big data typically need to have significant memory in order to store the large amounts of data from the different data sources for subsequent processing. The use of off-line or non-real-time processing usually significantly increases the cost of the computer systems and can degrade their performance, which in turn can degrade the user experience when using applications that leverage the data handled by the computer systems.
Note that like reference numerals refer to corresponding parts throughout the drawings. Moreover, multiple instances of the same part are designated by a common prefix separated from an instance number by a dash.